Decoding jointly coded transform type and subblock pattern information

ABSTRACT

Techniques and tools are described for decoding jointly coded information. For example, a decoder decodes a variable length code [“VLC”] signaled at macroblock level that jointly represents a transform type signal level, transform type, and subblock pattern. The decoder decodes one or more VLCs signaled at block level, each jointly representing a transform type and subblock pattern. The decoder may select between multiple VLC tables for the VLCs signaled macroblock level and/or block level.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/781,995, filed Feb. 4, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/051,094, filed Jul. 31, 2018, now U.S. Pat. No.10,554,985, which is a continuation of U.S. patent application Ser. No.15/068,325, filed Mar. 11, 2016, now U.S. Pat. No. 10,063,863, which isa continuation of U.S. patent application Ser. No. 12/815,029, filedJun. 14, 2010, now U.S. Pat. No. 9,313,509, which is a divisional ofU.S. patent application Ser. No. 10/893,168, filed Jul. 17, 2004, nowU.S. Pat. No. 7,738,554, the disclosure of which is incorporated hereinby reference, which claims the benefit of U.S. Provisional PatentApplication Ser. No. 60/488,710, filed Jul. 18, 2003, the disclosure ofwhich is incorporated herein by reference.

COPYRIGHT AUTHORIZATION

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by any one of the patentdisclosure, as it appears in the Patent and Trademark Office patentfiles or records, but otherwise reserves all copyright rightswhatsoever.

TECHNICAL FIELD

The invention relates generally to video and other digital media codingand decoding.

BACKGROUND

With the increased popularity of DVDs, music delivery over the Internet,and digital cameras, digital media have become commonplace. Engineersuse a variety of techniques to process digital audio, video, and imagesefficiently while still maintaining quality. To understand thesetechniques, it helps to understand how the audio, video, and imageinformation is represented and processed in a computer.

I. Representation of Media Information in a Computer

A computer processes media information as a series of numbersrepresenting that information. For example, a single number mayrepresent the intensity of brightness or the intensity of a colorcomponent such as red, green or blue for each elementary small region ofa picture, so that the digital representation of the picture consists ofone or more arrays of such numbers. Each such number may be referred toas a sample. For a color image, it is conventional to use more than onesample to represent the color of each elemental region, and typicallythree samples are used. The set of these samples for an elemental regionmay be referred to as a pixel, where the word “pixel” is a contractionreferring to the concept of a “picture element.” For example, one pixelmay consist of three samples that represent the intensity of red, greenand blue light necessary to represent the elemental region. Such a pixeltype is referred to as an RGB pixel. Several factors affect quality,including sample depth, resolution, and frame rate (for video).

Sample depth is a property normally measured in bits that indicates therange of numbers that can be used to represent a sample. When morevalues are possible for the sample, quality can be higher because thenumber can capture more subtle variations in intensity and/or a greaterrange of values. Images with higher resolution tend to look crisper thanother images and contain more discernable useful details. Video withhigher frame rate tends to mimic the smooth motion of natural objectsbetter than other video, and can similarly be considered to contain moredetail in the temporal dimension. For all of these factors, the tradeofffor high quality is the cost of storing and transmitting the informationin terms of the bit rate necessary to represent the sample depth,resolution and frame rate, as Table 1 shows.

TABLE 1 Bit rates for different quality levels of raw video Bit RateBits Per Pixel Resolution Frame Rate (in millions (sample depth times(in pixels, (in frames of bits per samples per pixel) Width × Height)per second) second) 8 (value 0-255, 160 × 120 7.5 1.2 monochrome) 24(value 0-255, RGB) 320 × 240 15 27.6 24 (value 0-255, RGB) 640 × 480 30221.2 24 (value 0-255, RGB) 1280 × 720  60 1327.1

Despite the high bit rate necessary for sending high quality video (suchas HDTV), companies and consumers increasingly depend on computers tocreate, distribute, and play back high quality content. For this reason,engineers use compression (also called source coding or source encoding)to reduce the bit rate of digital media. Compression decreases the costof storing and transmitting the information by converting theinformation into a lower bit rate form. Decompression (also calleddecoding) reconstructs a version of the original information from thecompressed form. A “codec” is an encoder/decoder system. Two categoriesof compression are lossless compression and lossy compression.

Lossless compression reduces the bit rate of information by removingredundancy from the information without any reduction in fidelity. Forexample, a series of ten consecutive pixels that are all exactly thesame shade of red could be represented as a code for the particularshade of red and the number ten as a “run length” of consecutive pixels,and this series can be perfectly reconstructed by decompression from thecode for the shade of red and the indicated number (ten) of consecutivepixels having that shade of red. Lossless compression techniques reducebit rate at no cost to quality, but can only reduce bit rate up to acertain point. Decreases in bit rate are limited by the inherent amountof variability in the statistical characterization of the input data,which is referred to as the source entropy. Entropy coding is anotherterm for lossless compression.

In contrast, with lossy compression, the quality suffers somewhat butthe achievable decrease in bit rate is more dramatic. For example, aseries of ten pixels, each being a slightly different shade of red, canbe approximated as ten pixels with exactly the same particularapproximate red color. Lossy compression techniques can be used toreduce bit rate more than lossless compression techniques, but some ofthe reduction in bit rate is achieved by reducing quality, and the lostquality cannot be completely recovered. Lossy compression is often usedin conjunction with lossless compression—in a system design in which thelossy compression establishes an approximation of the information andlossless compression techniques are applied to represent theapproximation. For example, the series of ten pixels, each a slightlydifferent shade of red, can be represented as a code for one particularshade of red and the number ten as a run-length of consecutive pixels.In decompression, the original series would then be reconstructed as tenpixels with the same approximated red color.

II. Quantization

According to one possible definition, quantization is a term used for anapproximating non-reversible mapping function commonly used for lossycompression, in which there is a specified set of possible outputvalues, and each member of the set of possible output values has anassociated set of input values that result in the selection of thatparticular output value. A variety of quantization techniques have beendeveloped, including scalar or vector, uniform or non-uniform, andadaptive or non-adaptive quantization.

A. Scalar Quantizers

According to one possible definition, a scalar quantizer is anapproximating functional mapping x→Q[x] of an input value x to aquantized value Q[x]. FIG. 1 shows a “staircase” I/O function (100) fora scalar quantizer. The horizontal axis is a number line for a realnumber input variable x, and the vertical axis indicates thecorresponding quantized values Q[x]. The number line is partitioned bythresholds such as the threshold (110). Each value of x within a givenrange between a pair of adjacent thresholds is assigned the samequantized value Q[x]. For example, each value of x within the range(120) is assigned the same quantized value (130). (At a threshold, oneof the two possible quantized values is assigned to an input x,depending on the system.) Overall, the quantized values Q[x] exhibit adiscontinuous, staircase pattern. The distance the mapping continuesalong the number line depends on the system, typically ending after afinite number of thresholds. The placement of the thresholds on thenumber line may be uniformly spaced (as shown in FIG. 1) ornon-uniformly spaced.

A scalar quantizer can be decomposed into two distinct stages. The firststage is the classifier stage, in which a classifier function mappingx→A[x] maps an input x to a quantization index A[x], which is ofteninteger-valued. In essence, the classifier segments an input number lineor data set. FIG. 2a shows a generalized classifier (200) and thresholdsfor a scalar quantizer. As in FIG. 1, a number line for a real numbervariable x is segmented by thresholds such as the threshold (210). Eachvalue of x within a given range such as the range (220) is assigned thesame quantized value Q[x]. FIG. 2b shows a numerical example of aclassifier (250) and thresholds for a scalar quantizer.

In the second stage, a reconstructor functional mapping k→β[k] maps eachquantization index k to a reconstruction value β[k]. In essence, thereconstructor places steps having a particular height relative to theinput number line segments (or selects a subset of data set values) forreconstruction of each region determined by the classifier. Thereconstructor functional mapping may be implemented, for example, usinga lookup table. Overall, the classifier relates to the reconstructor asfollows:Q[x]=β[A[x]]  (1).

The distortion introduced by using such a quantizer may be computed witha difference-based distortion measure d(x−Q[x]). Typically, such adistortion measure has the property that d(x−Q[x]) increases as x−Q[x]deviates from zero; and typically each reconstruction value lies withinthe range of the corresponding classification region, so that thestraight line that would be formed by the functional equation Q[x]=xwill pass through every step of the staircase diagram (as shown inFIG. 1) and therefore Q[Q[x]] will typically be equal to Q[x]. Ingeneral, a quantizer is considered better in rate-distortion terms ifthe quantizer results in a lower average value of distortion than otherquantizers for a given bit rate of output. More formally, a quantizer isconsidered better if, for a source random variable X, the expected(i.e., the average or statistical mean) value of the distortion measureD=E_(X){d(X−Q[X])} is lower for an equal or lower entropy H of A[X]. Themost commonly-used distortion measure is the squared error distortionmeasure, for which d(|x−y|)=|x−y|². When the squared error distortionmeasure is used, the expected value of the distortion measure (D) isreferred to as the mean squared error.

B. Dead Zone+Uniform Threshold Quantizers

According to one possible definition, a dead zone plus uniform thresholdquantizer [“DZ+UTQ”] is a quantizer with uniformly spaced thresholdvalues for all classifier regions except the one containing the zeroinput value (which is called the dead zone [“DZ”]). A DZ+UTQ has aclassifier index mapping rule x→A[x] that can be expressed based on twoparameters. FIG. 3 shows a staircase I/O function (300) for a DZ+UTQ,and FIG. 4a shows a generalized classifier (400) and thresholds for aDZ+UTQ. The parameter s, which is greater than 0, indicates the stepsize for all steps other than the DZ. Mathematically, all s_(i) areequal to s for i≠0. The parameter z, which is greater than or equal to0, indicates the ratio of the DZ size to the size of the other steps.Mathematically, s₀=z·s. In FIG. 4a , z is 2, so the DZ is twice as wideas the other classification zones. The index mapping rule x→A[x] for aDZ+UTQ can be expressed as:

$\begin{matrix}{{{A\lbrack x\rbrack} = {{{sign}(x)}*{\max\left( {0,\left\lfloor {\frac{x}{s} - \frac{z}{2} + 1} \right\rfloor} \right)}}},} & (2)\end{matrix}$where └·┘ denotes the smallest integer less than or equal to theargument and where sign(x) is the function defined as:

$\begin{matrix}{{{sign}(x)} = \left\{ {\begin{matrix}{{{+ 1},}\ } \\{{{- 1},}\ }\end{matrix}\begin{matrix}{{{{for}\mspace{14mu} x} \geq 0},} \\{{{for}\mspace{14mu} x} < 0.}\end{matrix}} \right.} & {(3).}\end{matrix}$

FIG. 4b shows a numerical example of a classifier (450) and thresholdsfor a DZ+UTQ with s=1 and z=2. FIGS. 1, 2 a, and 2 b show a special caseDZ+UTQ with z=1. Quantizers of the UTQ form have good performance for avariety of statistical sources. In particular, the DZ+UTQ form isoptimal for the statistical random variable source known as theLaplacian source.

In some system designs (not shown), an additional consideration may benecessary to fully characterize a DZ+UTQ classification rule. Forpractical reasons there may be a need to limit the range of values thatcan result from the classification function A[x] to some reasonablefinite range. This limitation is referred to as clipping. For example,in some such systems the classification rule could more precisely bedefined as:

$\begin{matrix}{{{A\lbrack x\rbrack} = {{{sign}(x)}*{\min\left\lbrack {g,{\max\left( {0,\left\lfloor {\frac{x}{s} - \frac{z}{2} + 1} \right\rfloor} \right)}} \right\rbrack}}},} & (4)\end{matrix}$where g is a limit on the absolute value of A[x]. In much of thetheoretical analysis presented herein, consideration of clipping isomitted as it unduly complicates the analysis without advancing theexplanation. Moreover, although the clipping shown in the above exampleis symmetric about zero, the clipping does not need to be symmetric, andoften is not exactly symmetric. For example, a common clipping rangewould be such that the value of A[x] is limited to some range from−2^(B) to +2^(B)−1 so that A[x] can be represented as an integer using atwo's complement representation that uses B+1 bits, where B+1 may beequal to 8 or 16 or another particular selected number of bits.

C. Reconstruction Rules

Different reconstruction rules may be used to determine thereconstruction value for each quantization index. These include theoptimal reconstruction rule and the single offset reconstruction rule(of which the mid-point reconstruction rule is an example). FIG. 5 showsreconstruction points according to different reconstruction rules for aparticular shape of a source probability distribution function f(x). Fora range of values between two thresholds t₁ and t_(j+1), thereconstruction value r_(j,mid) according to the mid-point reconstructionrule bisects the range (thus, r_(j,mid)=(t₁+t_(j+1))/2). For the exampleprobability distribution function shown in FIG. 5, this fails to accountfor the fact that values to the left of the mid-point are more likelythan values to the right of the mid-point. The reconstruction valuer_(j,opt) according to the optimal reconstruction rule accounts for theprobability distribution.

In general, a probability distribution function [“pdf”] indicates theprobabilities for the different values of a variable. One possibledefinition of the optimal reconstruction value r_(j,opt) for each regionbetween two neighboring thresholds t₁ and t_(j+1) for a pdf f(x) can beexpressed as:

$\begin{matrix}{r_{j,{opt}} = {{\min\limits_{y}}^{- 1}{\int\limits_{t_{j}}^{t_{j + 1}}{{d\left( {x - y} \right)}{f(x)}{{dx}.}}}}} & (5)\end{matrix}$Assuming that the pdf f(x) for a given source is symmetric around zero,one possible definition of the optimal reconstruction rule of a DZ+UTQfor a symmetric, difference-based distortion measure d(|x−y|) is:

$\begin{matrix}{{\beta\lbrack k\rbrack} = \left\{ \begin{matrix}{{{\min\limits_{y}}^{- 1}{\int^{\frac{zs}{2}}{\left\lbrack {{d\left( {{x - y}} \right)} + {d\left( {{y - x}} \right)}} \right\rbrack{f(x)}{dx}}}},} & {{{{for}\mspace{14mu} k} = 0},} \\{{{{sign}(k)}{{\min\limits_{y}}^{- 1}{\int_{{\frac{zs}{2} + {{({{k} - 1})}s}}\ }^{\frac{zs}{2} + {{k}s}}{{d\left( {{x - y}} \right)}{f(x)}{dx}}}}},} & {{{for}\mspace{14mu} k} \neq 0.}\end{matrix} \right.} & {(6),}\end{matrix}$where y is the quantized value Q[x], and where the rule finds thequantized value Q[x] that results in the smallest distortion accordingto the distortion measure. Typically, the optimal quantized value forβ[0] is equal to 0, and that will be assumed to be true for theremainder of this description. For minimizing mean squared error, theoptimal reconstruction rule sets the reconstruction value for eachregion equal to the conditional mean of the input values in that region.Stated more precisely, the optimal reconstruction value r_(j,opt) forthe region between two neighboring thresholds t₁ and t_(j+1) for a pdff(x) when using the mean squared error distortion measure is given by

$\begin{matrix}{r_{j,{opt}} = {\frac{\int\limits_{t_{j}}^{t_{j + 1}}{{x \cdot {f(x)}}{dx}}}{\int\limits_{t_{j}}^{t_{j + 1}}{{f(x)}{dx}}}.}} & (7)\end{matrix}$

According to one possible definition for a DZ+UTQ, the single-offsetreconstruction rule is based on an offset parameter Δ, where ordinarily0<Δ≤s/2, and the rule is:

$\begin{matrix}{{\beta\lbrack k\rbrack} = \left\{ \begin{matrix}{0,} & {{{{for}\mspace{14mu} k} = 0},} \\{{{{sign}(k)}\left\lbrack {{\left( {{k} + \frac{z}{2} - 1} \right)s} + \Delta} \right\rbrack},} & {{{for}\mspace{14mu} k} \neq 0.}\end{matrix} \right.} & {(8).}\end{matrix}$

The mid-point reconstruction rule is a special case of the single-offsetreconstruction rule, specified by Δ=s/2. Mid-point reconstruction iscommonly used for convenience due to its simplicity. And, in the limitas s becomes very small, the performance of the mid-point rule becomesoptimal under a variety of well-behaved mathematical conditions.

D. Specifying Reconstruction Values, Constructing Classifiers

Standards and product specifications that focus only on achievinginteroperability will often specify reconstruction values withoutnecessarily specifying the classification rule. In other words, somespecifications may define the functional mapping k→β[k] without definingthe functional mapping x→A[x]. This allows a decoder built to complywith the standard/specification to reconstruct information correctly. Incontrast, encoders are often given the freedom to change the classifierin any way that they wish, while still complying with thestandard/specification.

Numerous systems for adjusting quantization thresholds have beendeveloped. Many standards and products specify reconstruction valuesthat correspond to a typical mid-point reconstruction rule (e.g., for atypical simple classification rule) for the sake of simplicity. Forclassification, however, the thresholds can in fact be adjusted so thatcertain input values will be mapped to more common (and hence, lower bitrate) indices, which makes the reconstruction values closer to optimal.FIG. 6 shows such adjusted thresholds for a classifier (600). Theoriginal thresholds (such as old t_(j)) are situated halfway between thereconstruction points. The thresholds are moved outward on the numberline, away from 0. Before the adjustment, a marginal value (shownbetween the old t_(j) and the new t_(j)) is mapped to r_(j). After theadjustment, the marginal value is mapped to r₀. The decoder performsreconstruction without knowledge of the adjustments done in the encoder.

For optimal encoding, an encoder may adjust quantization thresholds tooptimally fit a given set of reconstruction values as follows. Theprobability p_(j) for the source random variable X to fall within arange j between t_(j) and t_(j+1) (where t_(j+1)>t_(j)) for a source pdff(x) is:

$\begin{matrix}{{p_{j} = {\int\limits_{t_{j}}^{t_{j + 1}}{{f(x)}{dx}}}},} & (9)\end{matrix}$and the number of bits necessary to represent an event with probabilityp_(j) in an ideal lossless communication system may be quantified as:

$\begin{matrix}{{h_{j} = {\log_{2}\frac{1}{p_{j}}}},} & (10)\end{matrix}$where the h_(j) is expressed in terms of bits. The total entropy of theclassifier is then given by

$\begin{matrix}{H = {\sum\limits_{j}{{p_{j} \cdot h_{j}}\;{{bits}.}}}} & (11)\end{matrix}$

In general, if the encoder is required to use b_(j) bits to indicate theselection of the reconstruction value r_(j), the encoder may evaluateand optimize its thresholds according to minimization of therate-distortion relation D+λR, where D indicates distortion, R indicatesbit usage, and λ is a tuning parameter for favoring a particularselected balance between distortion and bit rate. For each particularthreshold t_(j+1) between two points r_(j) and r_(j+1), the encoder canset t_(j+1) to the x that satisfies:d(x−r _(j))+λb _(j) =d(x−r _(j+1))+λb _(j+1)  (12).

In an ideal design, b_(j) will be approximately equal to h_(j), andmodern lossless coding techniques can be used to very nearly achievethis goal. In a design using some non-ideal lossless coding technique torepresent the output of the classifier, b_(j) may have some other value.

Note in summation that optimal decision thresholds can be selected usingequation (12), that optimal reconstruction values can be selected usingequation (5) or (7), and that optimal bit usage can be computed bysetting b_(j) equal to h_(j) as given by equation (10) or to the numberof bits used in some other lossless code (such as a Huffman codedesigned using equation (9) or a fixed-length code). In somehighly-optimized scalar quantizer system designs, reconstruction values(initially uniformly spaced) are analyzed to adjust thresholds inencoder analysis, then use of the adjusted thresholds is analyzed to setthe number of bits needed to represent the output of the classifierusing lossless coding and to set the reconstruction values in decoderanalysis. The new reconstruction values are then analyzed to adjustthresholds, and so on, until the thresholds and/or reconstruction valuesstabilize across iterations.

III. Compression and Decompression Systems

In general, video compression techniques include “intra-picture”compression and “inter-picture” compression, where a picture is, forexample, a progressively scanned video frame, an interlaced video frame(having alternating lines for video fields), or an interlaced videofield. For progressive frames, intra-picture compression techniquescompress individual frames (typically called I-frames or key frames),and inter-picture compression techniques compress frames (typicallycalled predicted frames, P-frames, or B-frames) with reference topreceding and/or following frames (typically called reference or anchorframes).

Both intra and inter-picture compression techniques often use areversible frequency transform operation, which generates a set offrequency domain (i.e., spectral) coefficients. For intra-picturecompression, the transform is typically applied to a block of samples.For inter-picture compression, the transform is typically applied to ablock of motion-compensation prediction residual information. A discretecosine transform [“DCT”] is a type of frequency transform. The resultingblocks of transform coefficients are quantized and entropy encoded. Adecoder typically entropy decodes and reconstructs transformcoefficients (e.g., DCT coefficients) that were quantized and performsan inverse frequency transform such as an IDCT.

A. Intra-Compression in Windows Media Video, Version 8 [“WMV8”]

Microsoft Corporation's Windows Media Video, Version 8 [“WMV8”] includesa video encoder and a video decoder. The WMV8 encoder uses intra-frameand inter-frame compression, and the WMV8 decoder uses intra-frame andinter-frame decompression.

FIG. 7 illustrates block-based intraframe compression (700) of a 8×8block (705) of samples in a frame in the WMV8 encoder. The WMV8 encoderhere splits a frame into 8×8 blocks of samples and applies an 8×8 DCT(710) to individual blocks such as the block (705). The encoderquantizes (720) the DCT coefficients (715), resulting in an 8×8 block ofquantized DCT coefficients (725). For example, the encoder applies auniform, scalar quantization step size to each coefficient.

Further encoding varies depending on whether a coefficient is a DCcoefficient, an AC coefficient in the top row or left column, or anotherAC coefficient. The encoder encodes the DC coefficient (726) as adifferential from the DC coefficient (736) of a neighboring 8×8 block,which is a previously encoded top or left neighbor block. The encoderentropy encodes (740) the differential. The entropy encoder can encodethe left column or top row of AC coefficients as differentials from acorresponding column or row of a neighboring 8×8 block. FIG. 7 shows theleft column (727) of AC coefficients encoded as differentials (747) fromthe left column (737) of the neighboring (actually situated to the left)block (735). The encoder scans (750) the 8×8 block (745) of predicted,quantized AC DCT coefficients into a one-dimensional array (755) andthen entropy encodes the scanned coefficients using a variation of runlength coding (760). The encoder selects an entropy code from one ormore run/level/last tables (765) and outputs the entropy code.

A WMV8 decoder (not shown) produces a reconstructed version of theoriginal block (705). The decoder determines the DC predictor for the DCcoefficient and decodes the DC differential. In particular, thefollowing pseudocode illustrates the DC differential decoding process inWMV8.

DCDifferential=vlc_decode( )

if (DCDifferential==ESCAPECODE)

-   -   DCDifferential=flc_decode(8)

DCSign=flc_decode(1)

if (DCSign==1)

-   -   DCDifferential=−DCDifferential

The WMV8 decoder combines the DC differential with the predictor for theDC coefficient to reconstruct the DC coefficient. The decoder entropydecodes the AC coefficients using one or more run/level/last tables, andscans the coefficients back into a two-dimensional array. The WMVdecoder computes a predictor for the top row or left column of ACcoefficients if appropriate. The decoder inverse quantizes thecoefficients and performs an IDCT.

While DC differential coding and decoding as in WMV8 provide goodperformance in many scenarios, there are opportunities for improvement.In particular, DC differential coding and decoding as in WMV8 are noteasily applied for smaller quantization sizes. This is because at thesmaller quantization sizes, VLC code table size for DC differentialsbecomes inefficiently large for many devices for practical applications.

B. Video Codec Standards

Various standards specify aspects of video decoders as well as formatsfor compressed video information. These standards include H.261, MPEG-1,H.262 (also called MPEG-2), H.263, and MPEG-4. Directly or byimplication, these standards may specify certain encoder details, butother encoder details are not specified. Different standards incorporatedifferent techniques, but each standard typically specifies some kind ofinverse frequency transform and entropy decoding. For information, seethe respective standard documents.

SUMMARY

In summary, the detailed description is directed to various techniquesand tools for video encoding and decoding. Some described tools andtechniques relate to coding of DC coefficients in video and otherdigital media coding. More particularly, the techniques and tools relateto signaling for DC coefficients at small quantization step sizes. Otherdescribed tools and techniques relate to other features of videoencoding and decoding. The techniques and tools can be used incombination or independently.

According to a first set of tools and techniques, a tool such as a videoencoder or decoder processes a first code that indicates a DCdifferential for a DC coefficient and a second code that indicates avalue refinement for the DC differential. For example, a video encoderencodes the DC coefficient based at least in part on the first andsecond codes. Or, a video decoder reconstructs the DC coefficient duringdecoding based at least in part on the first and second codes.

According to a second set of tools and techniques, a tool such as avideo encoder or decoder processes a VLC for a first DC differential fora first DC coefficient at a first quantization step size. The tool usesa VLC table that indicates DC differentials for DC coefficients at andabove a second quantization step size larger than the first quantizationstep size.

According to a third set of tools and techniques, a tool such as a videoencoder or decoder processes a code for a DC differential for a DCcoefficient, where the code is a FLC having a length that variesdepending on quantization step size. For example, the FLC indicates arefinement value for the DC differential. Or, when an escape code isused for the DC differential, the FLC indicates a value for the DCdifferential.

Additional features and advantages will be made apparent from thefollowing detailed description of various embodiments that proceeds withreference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a chart showing a staircase I/O function for a scalarquantizer according to the prior art.

FIGS. 2a and 2b are charts showing classifiers and thresholds for scalarquantizers according to the prior art.

FIG. 3 is a chart showing a staircase I/O function for a DZ+UTQaccording to the prior art.

FIGS. 4a and 4b are charts showing classifiers and thresholds forDZ+UTQs according to the prior art.

FIG. 5 is a chart showing reconstruction points for differentreconstruction rules for a given pdf shape according to the prior art.

FIG. 6 is a chart showing adjustments to a classifier for a scalarquantizer according to the prior art.

FIG. 7 is a block diagram showing block-based intra-compressionaccording to the prior art.

FIG. 8 is a block diagram of a suitable computing environment in whichseveral described embodiments may be implemented.

FIGS. 9 and 10 are block diagrams of a video encoder system and a videodecoder system, respectively, in conjunction with which severaldescribed embodiments may be implemented.

FIGS. 11A-11G are diagrams for different syntax layers of a bitstream.

FIGS. 12A-12C are tables for VLCs that jointly code transform type,transform type signal level, and subblock pattern.

FIGS. 13A-13C are tables for VLCs that jointly code transform type andsubblock pattern.

FIGS. 14A-14C are tables for VLCs that code subblock pattern.

FIG. 15 is a diagram showing different transform types.

FIG. 16 is a listing of DC differential decoding pseudocode.

DETAILED DESCRIPTION

Some described embodiments relate to techniques and tools for signalingDC coefficients at small quantization step sizes. Other described toolsand techniques relate to other features of video encoding and decoding.The various techniques and tools can be used in combination orindependently.

I. Computing Environment

FIG. 8 illustrates a generalized example of a suitable computingenvironment (800) in which several of the described embodiments may beimplemented. The computing environment (800) is not intended to suggestany limitation as to scope of use or functionality, as the techniquesand tools may be implemented in diverse general-purpose orspecial-purpose computing environments.

With reference to FIG. 8, the computing environment (800) includes atleast one processing unit (810) and memory (820). In FIG. 8, this mostbasic configuration (830) is included within a dashed line. Theprocessing unit (810) executes computer-executable instructions and maybe a real or a virtual processor. In a multi-processing system, multipleprocessing units execute computer-executable instructions to increaseprocessing power. The memory (820) may be volatile memory (e.g.,registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flashmemory, etc.), or some combination of the two. The memory (820) storessoftware (880) implementing an encoder and/or decoder with specialsignaling of DC coefficients at small quantization step sizes.

A computing environment may have additional features. For example, thecomputing environment (800) includes storage (840), one or more inputdevices (850), one or more output devices (860), and one or morecommunication connections (870). An interconnection mechanism (notshown) such as a bus, controller, or network interconnects thecomponents of the computing environment (800). Typically, operatingsystem software (not shown) provides an operating environment for othersoftware executing in the computing environment (800), and coordinatesactivities of the components of the computing environment (800).

The storage (840) may be removable or non-removable, and includesmagnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any othermedium which can be used to store information and which can be accessedwithin the computing environment (800). The storage (840) storesinstructions for the software (880) implementing the encoder and/ordecoder.

The input device(s) (850) may be a touch input device such as akeyboard, mouse, pen, or trackball, a voice input device, a scanningdevice, or another device that provides input to the computingenvironment (800). For audio or video encoding, the input device(s)(850) may be a sound card, video card, TV tuner card, or similar devicethat accepts audio or video input in analog or digital form, or a CD-ROMor CD-RW that reads audio or video samples into the computingenvironment (800). The output device(s) (860) may be a display, printer,speaker, CD-writer, or another device that provides output from thecomputing environment (800).

The communication connection(s) (870) enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,audio or video input or output, or other data in a modulated datasignal. A modulated data signal is a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia include wired or wireless techniques implemented with anelectrical, optical, RF, infrared, acoustic, or other carrier.

The techniques and tools can be described in the general context ofcomputer-readable media. Computer-readable media are any available mediathat can be accessed within a computing environment. By way of example,and not limitation, with the computing environment (800),computer-readable media include memory (820), storage (840),communication media, and combinations of any of the above.

The techniques and tools can be described in the general context ofcomputer-executable instructions, such as those included in programmodules, being executed in a computing environment on a target real orvirtual processor. Generally, program modules include routines,programs, libraries, objects, classes, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. The functionality of the program modules may be combined or splitbetween program modules as desired in various embodiments.Computer-executable instructions for program modules may be executedwithin a local or distributed computing environment.

II. Video Encoder and Decoder

FIG. 9 is a block diagram of a generalized video encoder system (900),and FIG. 10 is a block diagram of a video decoder system (1000), inconjunction with which various described embodiments may be implemented.

The relationships shown between modules within the encoder and decoderindicate the main flow of information in the encoder and decoder; otherrelationships are not shown for the sake of simplicity. In particular,FIGS. 9 and 10 usually do not show side information indicating theencoder settings, modes, tables, etc. used for a video sequence, frame,macroblock, block, etc. Such side information is sent in the outputbitstream, typically after entropy encoding of the side information. Theformat of the output bitstream can be a Windows Media Video version 9 orother format.

The encoder (900) and decoder (1000) are block-based and use a 4:2:0macroblock format, with each macroblock including four 8×8 luminanceblocks (at times treated as one 16×16 macroblock) and two 8×8chrominance blocks. Alternatively, the encoder (900) and decoder (1000)are object-based, use a different macroblock or block format, or performoperations on sets of pixels of different size or configuration than 8×8blocks and 16×16 macroblocks.

Depending on implementation and the type of compression desired, modulesof the encoder or decoder can be added, omitted, split into multiplemodules, combined with other modules, and/or replaced with like modules.In alternative embodiments, encoders or decoders with different modulesand/or other configurations of modules perform one or more of thedescribed techniques.

A. Video Encoder

FIG. 9 is a block diagram of a general video encoder system (900) thatcan perform joint entropy coding and bitstream formation operations forvariable-size transform information. The encoder system (900) receives asequence of video frames including a current frame (905), and producescompressed video information (995) as output. Particular embodiments ofvideo encoders typically use a variation or supplemented version of thegeneralized encoder (900).

The encoder system (900) compresses predicted frames and key frames. Forthe sake of presentation, FIG. 9 shows a path for key frames through theencoder system (900) and a path for forward-predicted frames. Many ofthe components of the encoder system (900) are used for compressing bothkey frames and predicted frames. The exact operations performed by thosecomponents can vary depending on the type of information beingcompressed.

A predicted frame (also called p-frame, b-frame for bi-directionalprediction, or inter-coded frame) is represented in terms of prediction(or difference) from one or more other frames. A prediction residual isthe difference between what was predicted and the original frame. Incontrast, a key frame (also called an I-frame or intra-coded frame) iscompressed without reference to other frames.

If the current frame (905) is a forward-predicted frame, a motionestimator (910) estimates motion of macroblocks or other sets of pixelsof the current frame (905) with respect to a reference frame, which is areconstructed previous frame (925) buffered in the frame store (920). Inalternative embodiments, the reference frame is a later frame or thecurrent frame is bi-directionally predicted. The motion estimator (910)can estimate motion by pixel, ½ pixel, ¼ pixel, or other increments, andcan switch the precision of the motion estimation on a frame-by-framebasis or other basis. The precision of the motion estimation can be thesame or different horizontally and vertically. The motion estimator(910) outputs as side information motion information (915) such asmotion vectors. A motion compensator (930) applies the motioninformation (915) to the reconstructed previous frame (925) to form amotion-compensated current frame (935). The prediction is rarelyperfect, however, and the difference between the motion-compensatedcurrent frame (935) and the original current frame (905) is theprediction residual (945). Alternatively, a motion estimator and motioncompensator apply another type of motion estimation/compensation.

For DC coefficients at small quantization step sizes, the encodersignals DC coefficients using a syntax and code tables such as thosedescribed below. In particular, the encoder uses the code tables andproduces an output bitstream in compliance with the syntax below.

A frequency transformer (960) converts the spatial domain videoinformation into frequency domain (i.e., spectral) data. For block-basedvideo frames, the frequency transformer (960) applies a DCT or variantof DCT to blocks of the pixel data or prediction residual data,producing blocks of DCT coefficients. Alternatively, the frequencytransformer (960) applies another conventional frequency transform suchas a Fourier transform or uses wavelet or subband analysis. Inembodiments in which the encoder uses spatial extrapolation (not shownin FIG. 9) to encode blocks of key frames, the frequency transformer(960) can apply a re-oriented frequency transform such as a skewed DCTto blocks of prediction residuals for the key frame. The frequencytransformer (960) applies an 8×8, 8×4, 4×8, or other size frequencytransforms (e.g., DCT) to prediction residuals for predicted frames.

A quantizer (970) then quantizes the blocks of spectral datacoefficients. The quantizer applies uniform, scalar quantization to thespectral data with a step-size that varies on a frame-by-frame basis orother basis. Alternatively, the quantizer applies another type ofquantization to the spectral data coefficients, for example, anon-uniform, vector, or non-adaptive quantization, or directly quantizesspatial domain data in an encoder system that does not use frequencytransformations. In addition to adaptive quantization, the encoder (900)can use frame dropping, adaptive filtering, or other techniques for ratecontrol.

If a given macroblock in a predicted frame has no information of certaintypes (e.g., no motion information for the macroblock and no residualinformation), the encoder (900) may encode the macroblock as a skippedmacroblock. If so, the encoder signals the skipped macroblock in theoutput bitstream of compressed video information (995).

When a reconstructed current frame is needed for subsequent motionestimation/compensation, an inverse quantizer (976) performs inversequantization on the quantized spectral data coefficients. An inversefrequency transformer (966) then performs the inverse of the operationsof the frequency transformer (960), producing a reconstructed predictionresidual (for a predicted frame) or reconstructed samples (for anintra-coded frame). If the frame (905) being encoded is an intra-codedframe, then the reconstructed samples form the reconstructed currentframe (not shown). If the frame (905) being encoded is a predictedframe, the reconstructed prediction residual is added to themotion-compensated predictions (935) to form the reconstructed currentframe. The frame store (920) buffers the reconstructed current frame foruse in predicting a next frame. In some embodiments, the encoder appliesa deblocking filter to the reconstructed frame to adaptively smoothdiscontinuities between the blocks of the frame.

The entropy coder (980) compresses the output of the quantizer (970) aswell as certain side information (e.g., motion information (915),spatial extrapolation modes, quantization step size). Typical entropycoding techniques include arithmetic coding, differential coding,Huffman coding, run length coding, LZ coding, dictionary coding, andcombinations of the above. The entropy coder (980) typically usesdifferent coding techniques for different kinds of information (e.g., DCcoefficients, AC coefficients, different kinds of side information), andcan choose from among multiple code tables within a particular codingtechnique.

The entropy coder (980) puts compressed video information (995) in thebuffer (990). A buffer level indicator is fed back to bit rate adaptivemodules. The compressed video information (995) is depleted from thebuffer (990) at a constant or relatively constant bit rate and storedfor subsequent streaming at that bit rate. Therefore, the level of thebuffer (990) is primarily a function of the entropy of the filtered,quantized video information, which affects the efficiency of the entropycoding. Alternatively, the encoder system (900) streams compressed videoinformation immediately following compression, and the level of thebuffer (990) also depends on the rate at which information is depletedfrom the buffer (990) for transmission.

Before or after the buffer (990), the compressed video information (995)can be channel coded for transmission over the network. The channelcoding can apply error detection and correction data to the compressedvideo information (995).

B. Video Decoder

FIG. 10 is a block diagram of a general video decoder system (1000). Thedecoder system (1000) receives information (1095) for a compressedsequence of video frames and produces output including a reconstructedframe (1005). Particular embodiments of video decoders typically use avariation or supplemented version of the generalized decoder (1000).

The decoder system (1000) decompresses predicted frames and key frames.For the sake of presentation, FIG. 10 shows a path for key framesthrough the decoder system (1000) and a path for forward-predictedframes. Many of the components of the decoder system (1000) are used fordecompressing both key frames and predicted frames. The exact operationsperformed by those components can vary depending on the type ofinformation being decompressed.

A buffer (1090) receives the information (1095) for the compressed videosequence and makes the received information available to the entropydecoder (1080). The buffer (1090) typically receives the information ata rate that is fairly constant over time, and includes a jitter bufferto smooth short-term variations in bandwidth or transmission. The buffer(1090) can include a playback buffer and other buffers as well.Alternatively, the buffer (1090) receives information at a varying rate.Before or after the buffer (1090), the compressed video information canbe channel decoded and processed for error detection and correction.

The entropy decoder (1080) entropy decodes entropy-coded quantized dataas well as entropy-coded side information (e.g., motion information(1015), spatial extrapolation modes, quantization step size), typicallyapplying the inverse of the entropy encoding performed in the encoder.Entropy decoding techniques include arithmetic decoding, differentialdecoding, Huffman decoding, run length decoding, LZ decoding, dictionarydecoding, and combinations of the above. The entropy decoder (1080)frequently uses different decoding techniques for different kinds ofinformation (e.g., DC coefficients, AC coefficients, different kinds ofside information), and can choose from among multiple code tables withina particular decoding technique.

If the frame (1005) to be reconstructed is a forward-predicted frame, amotion compensator (1030) applies motion information (1015) to areference frame (1025) to form a prediction (1035) of the frame (1005)being reconstructed. For example, the motion compensator (1030) uses amacroblock motion vector to find a macroblock in the reference frame(1025). A frame buffer (1020) stores previous reconstructed frames foruse as reference frames. The motion compensator (1030) can compensatefor motion at pixel, ½ pixel, ¼ pixel, or other increments, and canswitch the precision of the motion compensation on a frame-by-framebasis or other basis. The precision of the motion compensation can bethe same or different horizontally and vertically. Alternatively, amotion compensator applies another type of motion compensation. Theprediction by the motion compensator is rarely perfect, so the decoder(1000) also reconstructs prediction residuals.

When the decoder needs a reconstructed frame for subsequent motioncompensation, the frame store (1020) buffers the reconstructed frame foruse in predicting a next frame. In some embodiments, the encoder appliesa deblocking filter to the reconstructed frame to adaptively smoothdiscontinuities between the blocks of the frame.

An inverse quantizer (1070) inverse quantizes entropy-decoded data. Ingeneral, the inverse quantizer applies uniform, scalar inversequantization to the entropy-decoded data with a step-size that varies ona frame-by-frame basis or other basis. Alternatively, the inversequantizer applies another type of inverse quantization to the data, forexample, a non-uniform, vector, or non-adaptive inverse quantization, ordirectly inverse quantizes spatial domain data in a decoder system thatdoes not use inverse frequency transformations.

An inverse frequency transformer (1060) converts the quantized,frequency domain data into spatial domain video information. Forblock-based video frames, the inverse frequency transformer (1060)applies an IDCT or variant of IDCT to blocks of the DCT coefficients,producing pixel data or prediction residual data for key frames orpredicted frames, respectively. Alternatively, the frequency transformer(1060) applies another conventional inverse frequency transform such asa Fourier transform or uses wavelet or subband synthesis. In embodimentsin which the decoder uses spatial extrapolation (not shown in FIG. 10)to decode blocks of key frames, the inverse frequency transformer (1060)can apply a re-oriented inverse frequency transform such as a skewedIDCT to blocks of prediction residuals for the key frame. The inversefrequency transformer (1060) applies an 8×8, 8×4, 4×8, or other sizeinverse frequency transforms (e.g., IDCT) to prediction residuals forpredicted frames.

The decoder (1000) processes DC coefficient information whenquantization step sizes are small, for example, as described below.

III. Example Bitstream Syntax and Semantics

An example bitstream includes information for a sequence of compressedprogressive video frames or other pictures. The bitstream is organizedinto several hierarchical layers that are decoded by a decoder such asthe decoder (1000) of FIG. 10. The highest layer is the sequence layer,which has information for the overall sequence of frames. Additionally,each compressed video frame is made up of data that is structured intothree hierarchical layers. From top to bottom the layers are: picture,macroblock, and block.

FIG. 11A is a syntax diagram for the sequence layer (1100), whichincludes a sequence header (1110) followed by data for the picture layer(see FIG. 11B, FIG. 11C). The sequence header (1110) includes severalsequence-level elements that are processed by the decoder and used todecode the sequence, including a macroblock quantization (DQUANT)element (1111) and quantizer specifier (QUANTIZER) element (1112).DQUANT (1111) is a 2-bit field that indicates whether or not thequantization step size can vary within a frame. There are three possiblevalues for DQUANT. If DQUANT=0, then the only one quantization step size(i.e. the frame quantization step size) can be used per frame. IfDQUANT=1 or 2, then it is possible to quantize each of the macroblocksin the frame differently.

The QUANTIZER (1112) is a 2-bit fixed length code [“FLC”] field thatindicates the quantizer used for the sequence. The quantizer types areencoded according to the following Table 2.

TABLE 2 Quantizer Specification FLC Quantizer specification 00 Quantizerimplicitly specified at frame level 01 Quantizer explicitly specified atframe level 10 5 QP deadzone quantizer used for all frames 11 3 QPdeadzone quantizer used for all frames

The VSTRANSFORM field is a 1-bit field that indicates whethervariable-sized transform coding is enabled for the sequence. IfVSTRANSFORM=0, then variable-sized transform coding is not enabled. IfVSTRANSFORM=1, then variable-sized transform coding is enabled.

FIG. 11B is a syntax diagram for the picture layer (1120) for aprogressive intra-frame [“progressive I-frame”]. Syntax diagrams forother pictures, such as P-frames and B-frames have many similar syntaxelements. FIG. 11C is a syntax diagram for the picture layer (1121) fora progressive P-frame. The picture layer (1120, 1121) includes a pictureheader (1130, 1139) followed by data for the macroblock layer. Thepicture header (1130, 1139) includes several picture-level elements thatare processed by the decoder and used to decode the corresponding frame.Some of those elements are only present if their presence is signaled orimplied by a sequence-level element or a preceding picture-levelelement.

For example, the picture header (1130, 1139) includes an intra transformDCT table (DCTDCTAB) element (1137). This field is present in P picturesand baseline I pictures (X8IF=0). DCTDCTAB (1137) is a 1-bit field thatsignals which of two sets of VLC tables is used to decode the transformDC coefficients in intra-coded blocks. If DCTDCTAB=0, then the lowmotion VLC tables (one for luminance DC, one for chrominance DC) areused. If DCTDCTAB=1 then the high motion VLC tables (one for luminanceDC, one for chrominance DC) are used. The transform DC VLC tables arelisted below.

The picture header (1130, 1139) includes a picture quantizer index(PQINDEX) element (1131). PQINDEX (1131) is a 5-bit field that signalsthe quantizer scale index for the entire frame. It is present in allpicture types. If the implicit quantizer is used (signaled by sequencefield QUANTIZER=00, see Table 2 above) then PQINDEX specifies both thepicture quantizer scale (PQUANT) and the quantizer (3QP or 5QP deadzone)used for the frame. Table 3 shows how PQINDEX is translated to PQUANTand the quantizer for implicit mode.

TABLE 3 PQINDEX to PQUANT/Quantizer Deadzone Translation (ImplicitQuantizer) Quantizer PQINDEX PQUANT Deadzone 0 NA NA 1 1 3 QP 2 2 3 QP 33 3 QP 4 4 3 QP 5 5 3 QP 6 6 3 QP 7 7 3 QP 8 8 3 QP 9 6 5 QP 10 7 5 QP11 8 5 QP 12 9 5 QP 13 10 5 QP 14 11 5 QP 15 12 5 QP 16 13 5 QP 17 14 5QP 18 15 5 QP 19 16 5 QP 20 17 5 QP 21 18 5 QP 22 19 5 QP 23 20 5 QP 2421 5 QP 25 22 5 QP 26 23 5 QP 27 24 5 QP 28 25 5 QP 29 27 5 QP 30 29 5QP 31 31 5 QP

If the quantizer is signaled explicitly at the sequence or frame level(signaled by sequence field QUANTIZER=01, 10 or 11, see Table 2 above)then PQINDEX is translated to the picture quantizer step size PQUANT asindicated by Table 4.

TABLE 4 PQINDEX to PQUANT Translation (Explicit Quantizer) PQUANT PQUANT3QP 5QP PQINDEX Deadzone Deadzone 0 NA NA 1 1 1 2 2 1 3 3 1 4 4 2 5 5 36 6 4 7 7 5 8 8 6 9 9 7 10 10 8 11 11 9 12 12 10 13 13 11 14 14 12 15 1513 16 16 14 17 17 15 18 18 16 19 19 17 20 20 18 21 21 19 22 22 20 23 2321 24 24 22 25 25 23 26 26 24 27 27 25 28 28 26 29 29 27 30 30 29 31 3131

Alternatively, instead of the translation shown in Table 4, PQUANT isequal to PQINDEX for all values of PQINDEX from 1 through 31 when thequantizer is signaled explicitly at the sequence or frame level.

The picture header (1130, 1139) also includes a half QP step (HALFQP)element (1134) and picture quantizer type (PQUANTIZER) element (1135).HALFQP (1034) is a 1-bit field present if PQINDEX (1033) is less than orequal to 8. HALFQP (1134) allows the picture quantizer to be expressedin half step increments over the low PQUANT range. If HALFQP=1 then thepicture quantizer step size is PQUANT+½. If HALFQP=0 then the picturequantizer step size is PQUANT. Therefore, if the 3QP deadzone quantizeris used then half step sizes are possible up to PQUANT=9 (i.e.,PQUANT=1, 1.5, 2, 2.5 . . . 8.5, 9) and then only integer step sizes areallowable above PQUANT=9. For the 5QP deadzone quantizer, half stepsizes are possible up to PQUANT=7 (i.e., 1, 1.5, 2, 2.5 . . . 6.5, 7).

PQUANTIZER (1135) is a 1-bit field present in all frame types if thesequence level field QUANTIZER=01 (see Table 2 above). In this case, thequantizer used for the frame is specified by PQUANTIZER. If PQUANTIZER=0then the 5QP deadzone quantizer is used for the frame. If PQUANTIZER=1then the 3QP deadzone quantizer is used.

The picture header (1130) further includes a macroblock quantization(VOPDQUANT) field (1136). The VOPDQUANT field (1136) is present in aprogressive P frame (1121) when the sequence-header DQUANT field isnon-zero. VOPDQUANT (1136) may be used to adjust quantization step sizesfor macroblocks (e.g., macroblocks at one or more edges of a frame, oron a per macroblock basis). The syntax of VOPDQUANT (1136) is dependenton the picture type and the value of DQUANT, potentially includingDQUANTFRM, DQPROFILE, DQSBEDGE, DQDBEDGE, DQBILEVEL, PQDIFF, and ABSPQfields.

If DQUANT=1, there are four possibilities. (1) The macroblocks locatedon the boundary are quantized with a second quantization step size(ALTPQUANT) while the rest of the macroblocks are quantized with theframe quantization step size (PQUANT). (2) Macroblocks located on twoadjacent edges (signaled with DQDBEDGE) are quantized with ALTPQUANTwhile the rest of the macroblocks are quantized with PQUANT. (3)Macroblocks located on one edge (signaled with DQSBEDGE) are quantizedwith ALTPQUANT while the rest of the macroblocks are quantized withPQUANT. (4) Every single macroblock can be quantized differently.

In this case, each macroblock can select from two quantization steps(PQUANT or ALTPQUANT) or each macroblock can be arbitrarily quantizedusing any step size. If DQUANT=2, the macroblocks located on theboundary are quantized with ALTPQUANT while the rest of the macroblocksare quantized with PQUANT.

The DQUANTFRM field is a 1-bit value that is present only when DQUANT=1.If DQUANTFRM=0, then the current picture is only quantized with PQUANT.

The DQPROFILE field is a 2-bit value that is present only when DQUANT=1and DQUANTFRAME=1. It indicates where quantization step size is allowedto change within the current picture.

TABLE 5 Macroblock Quantization Profile (DQPROFILE) Code Table FLCLocation 00 all four edges 01 double edges 10 single edges 11 allmacroblocks

The DQSBEDGE field is a 2-bit value present when DQPROFILE=single edges.It indicates which edge will be quantized with ALTPQUANT (left, top,right, or bottom).

The DQDBEDGE field is a 2-bit value present when DQPROFILE=double edges.It indicates which edges will be quantized with ALTPQUANT (left and top,top and right, right and bottom, or bottom and left).

The DQBILEVEL field is a 1-bit value that is present when DQPROFILE=allmacroblocks. If DQBILEVEL=1, then each macroblock in the picture cantake one of two possible values (PQUANT or ALTPQUANT). If DQBILEVEL=0,then each macroblock in the picture can take on any quantization stepsize.

The PQDIFF field is a 3-bit field that encodes either the PQUANTdifferential or encodes an escape code. If PQDIFF does not equal 7, thenPQDIFF encodes the differential and the ABSPQ field does not follow inthe bitstream. In this case, ALTPQUANT=PQUANT+PQDIFF+1. If PQDIFF=7,then the ABSPQ field follows in the bitstream and ALTPQUANT is decodedas ALTPQUANT=ABSPQ. ABSPQ is present in the bitstream if PQDIFF=7. Inthis case, ABSPQ directly encodes the value of ALTPQUANT as described.

For additional detail about VOPDQUANT (1136), see U.S. patentapplication Ser. No. 10/623,195, filed Jul. 18, 2003.

In FIG. 11C, the macroblock-level transform type flag (TTMBF) field is1-bit field present only in P-picture headers and only if thesequence-level field VSTRANSFORM=1. If TTMBF=0, then the TTFRM field isalso present in the picture layer.

The frame-level transform type (TTFRM) field is a 2-bit field present inP-picture headers if VSTRANSFORM=1 and TTMBF=0. The TTFRM field isdecoded using a code table to select a transform type. For example, theFLC=00 indicates a transform type of 8×8 transform, the FLC=01 indicatesa transform type of 8×4 transform, the FLC=10 indicates a transform typeof 4×8 transform, and the FLC=11 indicates a transform type of 4×4transform.

FIG. 11D is a macroblock-layer (1140) bitstream syntax diagram forprogressive I-frames. The bitstream syntax for the macroblock layer ofP-pictures and B-pictures contain many elements in common. FIG. 11E is amacroblock-layer (1142) bitstream syntax diagram for progressiveP-frames, showing a macroblock header for a non-skipped macroblock in 1MV mode and a macroblock header for a non-skipped macroblock in 4 MVmode. Data for a macroblock consists of a macroblock header followed byblock-layer data.

In the macroblock layer headers, the MQDIFF field is a variable-sizedfield present if the picture-layer field DQPROFILE=all macroblocks. Thesyntax for the MQDIFF field depends on the DQBILEVEL field.

If DQBILEVEL=1, then MQDIFF is a 1-bit field and the ABMSQ field doesnot follow in the bitstream. If MQDIFF=0, then MQUANT=PQUANT (meaningthat PQUANT is used as the quantization step size for the currentmacroblock). If MQDIFF=1, then MQUANT=ALTPQUANT.

If DQBILEVEL=0, then MQDIFF is a 3-bit field. In this case, MQDIFFdecodes either to an MQUANT differential or to an escape code asfollows. If MQDIFF does not equal 7, then MQDIFF encodes thedifferential and the ABSMQ field does not follow in the bitstream. Inthis case, MQUANT=PQUANT+MQDIFF. If MQDIFF=7, then the ABSMQ fieldfollows in the bitstream, and MQUANT is decoded as MQUANT=ABSMQ.

In FIG. 11E, the macroblock-level transform type (TTMB) field is avariable-length field present in P-picture macroblocks if thepicture-layer field TTMBF=1. As shown in FIGS. 12A-12C, the TTMB fieldspecifies the transform type, the signal level, and the subblockpattern. If the signal type specifies macroblock mode, the transformtype decoded from the TTMB field is used to decode all coded blocks inthe macroblock. If the signal type signals block mode, then thetransform type decoded from the TTMB field is used to decode the firstcoded block in the macroblock, and the transform type of the remainingblocks is coded at the block level. If the transform type is 8×4 or 4×8,then the subblock pattern indicates the subblock pattern of the firstblock. The subblock pattern indicates which of 8×4 or 4×8 subblocks haveat least one non-zero coefficient.

The table used to decode the TTMB field depends on the value of PQUANT.For PQUANT less than or equal to 4, the VLC table (1201) in FIG. 12A isused. For PQUANT greater than 4 and less than or equal to 12, the VLCtable (1202) in FIG. 12B is used. For PQUANT greater than 12, the VLCtable (1203) in FIG. 12C is used.

FIG. 11F is an intra-coded block-layer (1160) bitstream syntax diagram.The block-layer data includes a transform DC coefficient (DCCOEF)element (1161), an escape transform DC coefficient (DCCOEFESC) element(1162), and a transform DC sign (DCSIGN) element (1163).

The DCCOEF (1161) field is only present in intra-coded blocks. This is avariable-length codeword that encodes a transform DC differential. Thetransform DC decoding process is described further below. One of twosets of code tables is used to encode the DC differentials (the table issignaled in the DCTDCTAB (1137) field in the picture header as describedabove). The DC VLC tables are also listed below.

The DCCOEFESC (1162) field is only present in intra-coded blocks andonly if DCCOEF decodes to the escape code. The size of DCCOEFESC fieldcan be 8, 9 or 10 bits depending on the quantization step size of theblock.

DCSIGN (1163) is a 1-bit value that indicates the sign of the DCdifferential. If DCSIGN=0 then the DC differential is positive. IfDCSIGN=1 then the DC differential is negative.

FIG. 11G is an inter-coded block-layer (1165) bitstream syntax diagram.The block-level transform type (TTBLK) field is a variable-size field.The TTBLK field is present only in inter-coded blocks and only if themacroblock-level field TTMB indicates that the signaling level is Block.8×8 error blocks can be transformed using an 8×8 transform, two 8×4transforms, two 4×8 transforms, or four 4×4 transforms. The TTBLK fieldcodes the transform type for the block as well as the subblock patternif the transform type is 8×4 or 4×8. The table used to decode the TTBLKfield depends on the value of PQUANT. If PQUANT is less than or equal to4, the VLC table (1301) in FIG. 13A is used. If PQUANT is greater than 4and less than or equal to 12, the VLC table (1302) in FIG. 13B is used.If PQUANT is greater than 12, the VLC table (1303) in FIG. 13C is used.The TTBLK field is not present for the first block in each macroblocksince the transform type and subblock pattern decoded in TTMB are usedfor the first block. TTBLK is present for each coded block after thefirst. The subblock pattern indicates which of 8×4 or 4×8 subblocks haveat least one non-zero coefficient.

In FIG. 11G, the transform subblock pattern (SUBBLKPAT) field is avariable-size field. The SUBBLKPAT field is only present in inter-codedblocks and only if the transform type for the block is 8×4, 4×8, or 4×4.

For the 4×4 transform type, the SUBBLKPAT field indicates which of thefour 4×4 subblocks have at least one non-zero coefficient. The subblockpattern is coded as a 4-bit field, where each bit indicates whether thecorresponding subblock contains at least one non-zero coefficient. Forexample, when an 8×8 block has four 4×4 subblocks SB0, SB1, SB2, andSB3, SB0 is the top-left 4×4 subblock, SB1 is the top-right 4×4subblock, SB2 is the bottom-left 4×4 subblock, and SB3 is thebottom-right 4×4 subblock. The subblock pattern is coded as follows:subblock pattern=8*SB0+4*SB1+2*SB2+SB3,where SBx=0 if the corresponding subblock does not contain any non-zerocoefficients, and SBx=1 if the corresponding subblock contains at leastone non-zero coefficient.

FIGS. 14A-14C show VLC codes used to encode the subblock pattern. Thetable used depends on the value of PQUANT. If PQUANT is less than orequal to 4, the VLC table (1401) in FIG. 14A is used. If PQUANT isgreater than 4 and less than or equal to 12, the VLC table (1402) inFIG. 14B is used. If PQUANT is greater than 12, the VLC table (1403) inFIG. 14C is used.

For 8×4 or 4×8 transform types, the SUBBLKPAT field specifies which ofthe two subblocks have at least one non-zero coefficient. The data isencoded with the VLCs shown in the following table, in which an Xindicates that the subblock contains at least one non-zero coefficient.

VLC table for 8 × 4 and 4 × 8 subblock pattern codes SUBBLKPAT 8 × 4Subblock Pattern 4 × 8 Subblock Pattern VLC Top Bottom Left Right  0 X X10 X X X X 11 X XIV. Example Decoding and Dequantization

For typical blocks, a decoder such as the decoder (1000) of FIG. 10decodes coefficients, performs inverse quantization, and performs aninverse transform.

A. Decoding DC Differentials for Intra-Coded Blocks

The DC coefficient is coded differentially with respect to analready-decoded DC coefficient neighbor. This section describes theprocess used to decode the bitstream to obtain the DC differential.

FIG. 11F shows the bitstream elements used to encode/decode the DCdifferential. DCCOEF is decoded using one of two sets of VLC tables (onefor low motion and one for high motion). Each set of VLC tables includesa table for DC differentials for luminance blocks and a table for DCdifferentials for chrominance blocks. The table is specified by theDCTDCTAB (1137) field in the picture header. Based on the value ofDCTDCTAB, one of the VLC tables listed below is used to decode DCCOEF.This will yield either:

1) zero, or

2) the absolute value of the DC differential, or

3) the escape code.

If DCCOEF decodes to zero, the value of the DC differential is alsozero. Otherwise, further decoding is done to determine the value of theDC differential. If DCCOEF decodes to the escape code, the absolutevalue of the DC differential is encoded in the DCCOEFESC field. The sizeof the DCCOEFESC field is 8, 9 or 10 bits depending on the quantizationstep size of the block. The sign of the DC differential is obtained fromthe DCSIGN field. FIG. 16 lists pseudocode to illustrate the DCdifferential decoding process.

B. DC Differential VLC Tables for Intra-Coded Blocks

1. Low-Motion VLC Tables

TABLE 7 Low-motion Luminance DC Differential VLC Table DC VLC VLCDifferential Codeword Size 0 1 1 1 1 2 2 1 4 3 1 5 4 5 5 5 7 5 6 8 6 712 6 8 0 7 9 2 7 10 18 7 11 26 7 12 3 8 13 7 8 14 39 8 15 55 8 16 5 9 1776 9 18 108 9 19 109 9 20 8 10 21 25 10 22 155 10 23 27 10 24 154 10 2519 11 26 52 11 27 53 11 28 97 12 29 72 13 30 196 13 31 74 13 32 198 1333 199 13 34 146 14 35 395 14 36 147 14 37 387 14 38 386 14 39 150 14 40151 14 41 384 14 42 788 15 43 789 15 44 1541 16 45 1540 16 46 1542 16 473086 17 48 197581 23 49 197577 23 50 197576 23 51 197578 23 52 197579 2353 197580 23 54 197582 23 55 197583 23 56 197584 23 57 197585 23 58197586 23 59 197587 23 60 197588 23 61 197589 23 62 197590 23 63 19759123 64 197592 23 65 197593 23 66 197594 23 67 197595 23 68 197596 23 69197597 23 70 197598 23 71 197599 23 72 197600 23 73 197601 23 74 19760223 75 197603 23 76 197604 23 77 197605 23 78 197606 23 79 197607 23 80197608 23 81 197609 23 82 197610 23 83 197611 23 84 197612 23 85 19761323 86 197614 23 87 197615 23 88 197616 23 89 197617 23 90 197618 23 91197619 23 92 197620 23 93 197621 23 94 197622 23 95 197623 23 96 19762423 97 197625 23 98 197626 23 99 197627 23 100 197628 23 101 197629 23102 197630 23 103 197631 23 104 395136 24 105 395137 24 106 395138 24107 395139 24 108 395140 24 109 395141 24 110 395142 24 111 395143 24112 395144 24 113 395145 24 114 395146 24 115 395147 24 116 395148 24117 395149 24 118 395150 24 ESCAPE 395151 24

TABLE 8 Low-motion Chroma DC Differential VLC Table DC VLC VLCDifferential Codeword Size 0 0 2 1 1 2 2 5 3 3 9 4 4 13 4 5 17 5 6 29 57 31 5 8 33 6 9 49 6 10 56 6 11 51 6 12 57 6 13 61 6 14 97 7 15 121 7 16128 8 17 200 8 18 202 8 19 240 8 20 129 8 21 192 8 22 201 8 23 263 9 24262 9 25 406 9 26 387 9 27 483 9 28 482 9 29 522 10 30 523 10 31 1545 1132 1042 11 33 1043 11 34 1547 11 35 1041 11 36 1546 11 37 1631 11 381040 11 39 1629 11 40 1630 11 41 3256 12 42 3088 12 43 3257 12 44 617913 45 12357 14 46 24713 15 47 49424 16 48 3163208 22 49 3163209 22 503163210 22 51 3163211 22 52 3163212 22 53 3163213 22 54 3163214 22 553163215 22 56 3163216 22 57 3163217 22 58 3163218 22 59 3163219 22 603163220 22 61 3163221 22 62 3163222 22 63 3163223 22 64 3163224 22 653163225 22 66 3163226 22 67 3163227 22 68 3163228 22 69 3163229 22 703163230 22 71 3163231 22 72 3163232 22 73 3163233 22 74 3163234 22 753163235 22 76 3163236 22 77 3163237 22 78 3163238 22 79 3163239 22 803163240 22 81 3163241 22 82 3163242 22 83 3163243 22 84 3163244 22 853163245 22 86 3163246 22 87 3163247 22 88 3163248 22 89 3163249 22 903163250 22 91 3163251 22 92 3163252 22 93 3163253 22 94 3163254 22 953163255 22 96 3163256 22 97 3163257 22 98 3163258 22 99 3163259 22 1003163260 22 101 3163261 22 102 3163262 22 103 3163263 22 104 6326400 23105 6326401 23 106 6326402 23 107 6326403 23 108 6326404 23 109 632640523 110 6326406 23 111 6326407 23 112 6326408 23 113 6326409 23 1146326410 23 115 6326411 23 116 6326412 23 117 6326413 23 118 6326414 23ESCAPE 6326415 23

2. High-Motion Tables

TABLE 9 High-motion Luminance DC Differential VLC Table DC VLC VLCDifferential Codeword Size 0 2 2 1 3 2 2 3 3 3 2 4 4 5 4 5 1 5 6 3 5 7 85 8 0 6 9 5 6 10 13 6 11 15 6 12 19 6 13 8 7 14 24 7 15 28 7 16 36 7 174 8 18 6 8 19 18 8 20 50 8 21 59 8 22 74 8 23 75 8 24 11 9 25 38 9 26 399 27 102 9 28 116 9 29 117 9 30 20 10 31 28 10 32 31 10 33 29 10 34 4311 35 61 11 36 413 11 37 415 11 38 84 12 39 825 12 40 824 12 41 829 1242 171 13 43 241 13 44 1656 13 45 242 13 46 480 14 47 481 14 48 340 1449 3314 14 50 972 15 51 683 15 52 6631 15 53 974 15 54 6630 15 55 136416 56 1951 16 57 1365 16 58 3901 17 59 3895 17 60 3900 17 61 3893 17 627789 18 63 7784 18 64 15576 19 65 15571 19 66 15577 19 67 31140 20 68996538 25 69 996532 25 70 996533 25 71 996534 25 72 996535 25 73 99653625 74 996537 25 75 996539 25 76 996540 25 77 996541 25 78 996542 25 79996543 25 80 1993024 26 81 1993025 26 82 1993026 26 83 1993027 26 841993028 26 85 1993029 26 86 1993030 26 87 1993031 26 88 1993032 26 891993033 26 90 1993034 26 91 1993035 26 92 1993036 26 93 1993037 26 941993038 26 95 1993039 26 96 1993040 26 97 1993041 26 98 1993042 26 991993043 26 100 1993044 26 101 1993045 26 102 1993046 26 103 1993047 26104 1993048 26 105 1993049 26 106 1993050 26 107 1993051 26 108 199305226 109 1993053 26 110 1993054 26 111 1993055 26 112 1993056 26 1131993057 26 114 1993058 26 115 1993059 26 116 1993060 26 117 1993061 26118 1993062 26 ESCAPE 1993063 26

TABLE 10 High-motion Chroma DC Differential VLC Table DC VLC VLCDifferential Codeword Size 0 0 2 1 1 2 2 4 3 3 7 3 4 11 4 5 13 4 6 21 57 40 6 8 48 6 9 50 6 10 82 7 11 98 7 12 102 7 13 166 8 14 198 8 15 207 816 335 9 17 398 9 18 412 9 19 669 10 20 826 10 21 1336 11 22 1596 11 231598 11 24 1599 11 25 1654 11 26 2675 12 27 3194 12 28 3311 12 29 534913 30 6621 13 31 10696 14 32 10697 14 33 25565 15 34 13240 14 35 1324114 36 51126 16 37 25560 15 38 25567 15 39 51123 16 40 51124 16 41 5112516 42 25566 15 43 51127 16 44 51128 16 45 51129 16 46 102245 17 47204488 18 48 13087304 24 49 13087305 24 50 13087306 24 51 13087307 24 5213087308 24 53 13087309 24 54 13087310 24 55 13087311 24 56 13087312 2457 13087313 24 58 13087314 24 59 13087315 24 60 13087316 24 61 1308731724 62 13087318 24 63 13087319 24 64 13087320 24 65 13087321 24 6613087322 24 67 13087323 24 68 13087324 24 69 13087325 24 70 13087326 2471 13087327 24 72 13087328 24 73 13087329 24 74 13087330 24 75 1308733124 76 13087332 24 77 13087333 24 78 13087334 24 79 13087335 24 8013087336 24 81 13087337 24 82 13087338 24 83 13087339 24 84 13087340 2485 13087341 24 86 13087342 24 87 13087343 24 88 13087344 24 89 1308734524 90 13087346 24 91 13087347 24 92 13087348 24 93 13087349 24 9413087350 24 95 13087351 24 96 13087352 24 97 13087353 24 98 13087354 2499 13087355 24 100 13087356 24 101 13087357 24 102 13087358 24 10313087359 24 104 26174592 25 105 26174593 25 106 26174594 25 107 2617459525 108 26174596 25 109 26174597 25 110 26174598 25 111 26174599 25 11226174600 25 113 26174601 25 114 26174602 25 115 26174603 25 116 2617460425 117 26174605 25 118 26174606 25 ESCAPE 26174607 25

C. Computing DC Predictors for Intra-Coded Blocks

The quantized DC value for a current block is obtained by adding a DCpredictor to the DC differential. The DC predictor is obtained from oneof the previously decoded adjacent blocks, which may be labeledcandidate predictors A (from the block immediately above and to the leftof the current block), B (from the block immediately above the currentblock), and C (from the block immediately to the left of the currentblock). The values for A, B and C are the quantized DC values for therespective adjacent blocks.

In some cases there are missing adjacent blocks. If the current block isin the first block row of the frame, there are no A or B (and possiblyno C). If the current block is in the first block column in the frame,there are no A and C (and possibly no B) blocks. For these cases the DCpredictor is set to:DCPredictor=(1024+(DCStepSize>>1))/DCStepSize,where DCStepSize is a value described below.

Otherwise, a prediction direction is formed based on the values of A, Band C, and either the B or C predictor is chosen. The predictiondirection is calculated as follows. If the absolute value of (A−B) isless than or equal to the absolute value of (A−C), then the predictionis made from the left (C is the predictor). Otherwise, the prediction ismade from the top (B is the predictor).

The quantized DC coefficient is then calculated by adding the DCdifferential and the DC predictor as follows:DC CoeffQ=DCPredictor+DCDifferential

D. AC Coefficient Reconstruction For Intra-Coded Blocks

Non-zero AC coefficients are reconstructed using run-level decoding.Decoding run-level pairs produces a one-dimensional array of quantizedAC coefficients. The elements in the array are scanned out into atwo-dimensional array. If the ACPRED field in the macroblock layerspecifies that AC prediction is used for the blocks, then the top row orleft column of AC coefficients in the decoded block are treated asdifferential values from the coefficients in the corresponding row orcolumn in a predicted block. The predicted block is either the blockimmediately above or to the left of the current block.

E. Inverse-Quantization for Baseline I-Frame Pictures

In each macroblock of a picture (frame), the decoder decodes a DCcoefficient and set of AC coefficients, which were each quantized at theencoder. These quantized transform coefficients are dequantized for abaseline I-Frame picture as described below.

1. DC Inverse-Quantization

The quantized DC coefficient (DCCoeffQ) is reconstructed by performingthe following de-quantization operation:DCCoefficient=DC CoeffQ*DCStepSize

The value of DCStepSize is based on the value of PQUANT as follows:

For PQUANT equal to 1 or 2:

-   -   DCStepSize=2*PQUANT

For PQUANT equal to 3 or 4:

-   -   DCStepSize=8

For PQUANT greater than or equal to 5:

-   -   DCStepSize=PQUANT/2+6

2. Inverse AC Coefficient Quantization

AC coefficients are separately decoded. Depending on whether the 3-QP or5-QP deadzone quantizer is used, the non-zero quantized AC coefficientsare inverse quantized according to the following formula:dequant_coeff=quant_coeff*double_quant (if 3-QP deadzone quantizer), ordequant_coeff=quant_coeff*double_quant+sign(quant_coeff)*quant scale (if5-QP deadzone quantizer)

where:

quant_coeff is the quantized coefficient

dequant_coeff is the inverse quantized coefficient

double_quant=2*PQUANT+HalfStep

quant scale=PQUANT

PQUANT is encoded in the picture layer as described above. HalfStep isencoded in the picture layer via the HALFQP element as described above.

F. Determining Transform Type and Subblock Patterns for Inter-CodedBlocks of P Pictures

The macroblock-level transform type flag (TTMBF) field is a 1-bit fieldthat signals whether transform type coding is enabled at the frame ormacroblock level. If TTMBF=0, then the same transform type is used forall blocks in the frame. In this case, the transform type is signaled inthe TTFRM field that follows. If TTMBF=1, then the transform type canvary throughout the frame and is signaled at the macroblock or blocklevels.

The frame-level transform type (TTFRM) field is a variable-length fieldthat is present in the picture layer if TTMBF=0. TTFRM signals thetransform type used to transform the 8×8 pixel error signal in predictedblocks. The 8×8 error blocks can be transformed using an 8×8 transform,two 8×4 transforms, two 4×8 transforms, or four 4×4 transforms.

The macroblock-level transform type (TTMB) field is present only inInter macroblocks. TTMB encodes the transform type, the signaling mode,and the transform subblock pattern. If the signaling mode is macroblocksignaling, then the transform type decoded from the TTMB field is thesame for all blocks in the macroblock. If the transform type is 8×4 or4×8, then a subblock pattern is also decoded from the TTMB field. Inthis case, the subblock pattern applies to the first coded block in themacroblock, and the subblock patterns for all the blocks after the firstblock are coded in SUBBLKPAT fields at the block level. If the transformtype is 4×4, then the subblock patterns for the blocks are encoded inSUBBLKPAT fields at the block level.

If the signaling mode is block signaling, then the transform typedecoded from the TTMB field is applied to the first coded block in themacroblock, and the TTBLK field is not present for the first codedblock. For each of the remaining coded blocks, the TTBLK field indicatesthe transform type for that block. If the TTMB field indicates that thefirst transform type is 8×4 or 4×8, then a subblock pattern is alsodecoded from the TTMB field. In this case, the subblock pattern appliesto the first coded block in the macroblock.

The steps to reconstruct an inter-coded block include: 1) transform typeselection, 2) subblock pattern decode, 3) coefficient decode, 4) inversetransform, 5) obtain predicted block and 6) motion compensation (addpredicted and error blocks). As shown in FIG. 15, if variable-sizedtransform coding is enabled (signaled by the sequence-level fieldVSTRANSFORM=1), then an 8×8 error block can be transformed using one 8×8transform, or divided vertically and transformed with two 8×4transforms, or divided horizontally and transformed with two 4×8transforms, or divided into four quadrants and transformed with four 4×4transforms. The transform type is signaled at the picture, macroblock orblock level. As shown in the VLC tables (1201-1203) of FIGS. 12A-12C, ifTTMB indicates that the signal level is Block, then the transform typeis signaled at the block level for blocks after the first block of amacroblock. If the transform type is specified at the block level, thenthe TTBLK field is present within the bitstream as shown in FIG. 11G.The TTBLK field indicates the transform type used for a block. FIGS.13A-13C show VLC tables (1301-1303) used to encode the transform type ifblock mode signaling is used.

If variable-sized transform coding is not enabled, then the 8×8transform is used for all blocks.

If the transform type is 8×4, 4×8 or 4×4, then the decoder needsinformation about which of the subblocks have nonzero coefficients. For8×4 and 4×8 transform types, the subblock pattern is decoded as part ofthe TTMB or TTBLK field. If the transform type is 4×4, then theSUBBLKPAT field is present in the bitstream as shown in FIG. 11G.

If the subblock pattern indicates that no non-zero coefficients arepresent for the subblock, then no other information for that subblock ispresent in the bitstream. For the 8×4 transform type, the data for thetop subblock (if present) is coded first followed by the bottom subblock(if present). For the 4×8 transform type, the data for the left subblock(if present) is coded first followed by the right subblock (if present).For the 4×4 transform type, the data for the upper left subblock iscoded first followed, in order, by the upper right, lower left and lowerright subblocks (for any subblock with data present).

G. Coefficient Reconstruction for Blocks of P Pictures

The first step in reconstructing an inter-coded block is to reconstructthe transform coefficients. The process for decoding the bitstream toobtain run, level, and last flags for each non-zero coefficient in ablock or subblock is nearly identical to the process described fordecoding AC coefficients in intra-coded blocks, but the DC coefficientin not differentially coded (no distinction is made between the DC andAC coefficients; all coefficients are decoded using the same method).The process for decoding run-level pairs obtained in the coefficientdecoding process is nearly the same as the process described fordecoding coefficients in intra-coded blocks. The difference is that allcoefficients are run-level encoded (not just the AC coefficients as inintra-coded blocks). The one-dimensional array of quantized coefficientsproduced in the run-level decode process are scanned out into atwo-dimensional array.

H. Inverse-Quantization For P Pictures

The non-zero quantized transform coefficients are inverse quantized inone of two ways, depending on the value of PQUANT.

If the 3QP deadzone quantizer is used, the following formula describesthe inverse quantization process:dequant_coeff=quant_coeff*(2*quant_scale+halfstep).If the 5QP deadzone quantizer is used, the following formula describesthe inverse quantization process:dequant_coeff=quant_coeff*(2*quant_scale+halfstep)+sign(quant_coeff)*quant_scale,

where:

quant_coeff is the quantized coefficient

dequant_coeff is the inverse quantized coefficient

quant_scale=the quantizer scale for the block (either PQUANT or MQUANT)

halfstep=the half step encoded in the picture layer as described above.PQUANT is encoded in the picture layer as described above. MQUANT isencoded in the macroblock layer as described above.

V. Signaling for DC Coefficients with Small Quantization Step Sizes,Theory

In the implementation described in detail above, the range fordifferential DC coefficients becomes larger as the quantization stepsize becomes smaller. For example, the range for a quantization stepsize of 2 is twice as large as the range for a quantization step size of4. Further, the range for quantization step size of 1 is four times therange for quantization step size of 4. A VLC table used to directlyencode/decode the differential DC coefficient for such small step sizeswould need to be very large and would impose excessive memoryrequirements in some cases (e.g., in small footprint devices). Further,as the lowest quantization step sizes are infrequently or rarely used inpractical encoding scenarios, the cost of this additional memoryrequirement would not be justified.

The problem of excessively large VLC tables for very small quantizationsizes is addressed in this implementation by designing the VLC tables toaccommodate the range of differential DC coefficients when thequantization step size is 4. Then, for smaller quantization step sizes(e.g., 1 and 2), a multistage approach is used to signal thedifferential DC coefficient. More specifically, at a quantization stepsize of 2, a standard VLC table is used to decode a base VLC for thedifferential DC coefficient. An additional 1-bit code is also decoded,and this is used to refine the value of the differential DC coefficient.At a quantization step size of 1, a standard VLC table again is used todecode a base VLC for the differential DC coefficient. An additional2-bit code is also decoded and used to refine the value of thedifferential DC coefficient.

When the base VLC represents the escape code, a further FLC is used tosignal the differential DC coefficient. The size of the FLC changes withthe quantization step size. For example, the FLC is 8 bits forquantization steps sizes over 2, and 9 and 10 bits for quantization stepsizes of 2 and 1, respectively. This reduces bit rate for the escapecode FLCs for higher quantization step sizes.

Having described and illustrated the principles of our invention, itwill be recognized that the various embodiments can be modified inarrangement and detail without departing from such principles. It shouldbe understood that the programs, processes, or methods described hereinare not related or limited to any particular type of computingenvironment, unless indicated otherwise. Various types of generalpurpose or specialized computing environments may be used with orperform operations in accordance with the teachings described herein.Elements of embodiments shown in software may be implemented in hardwareand vice versa.

In view of the many possible embodiments to which the principles of ourinvention may be applied, we claim as our invention all such embodimentsas may come within the scope and spirit of the following claims andequivalents thereto.

We claim:
 1. In a computer system that implements a video decoder, amethod comprising: receiving, as part of encoded data in a bit streamfor at least part of a video sequence, a variable length code (VLC) fora macroblock with plural blocks, the VLC jointly representing atransform type signal level, a transform type, and a subblock pattern;and decoding at least part of the encoded data to reconstruct themacroblock, the plural blocks of the macroblock including a first blockthat is coded, and the first block including plural subblocks, includingdecoding the VLC to determine the transform type signal level, thetransform type, and the subblock pattern, wherein: the transform typeindicates a transform size that applies for the plural subblocks of thefirst block, the transform type further indicating how the first blockis split into the plural subblocks; and the subblock pattern indicateswhich of the plural subblocks of the first block have at least onenon-zero coefficient.
 2. The method of claim 1, wherein the transformtype applies for only the first block and the subblock pattern is forthe first block.
 3. The method of claim 1, wherein the transform typesignal level is macroblock level, the transform type applies for each ofthe plural blocks of the macroblock that is coded, and the subblockpattern is for the first block.
 4. The method of claim 3, wherein theencoded data further includes one or more other VLCs that each representa subblock pattern for one of the plural blocks of the macroblock thatis coded other than the first block.
 5. The method of claim 1, whereinthe transform type signal level is block level, the transform typeapplies for only the first block, and the subblock pattern is for thefirst block.
 6. The method of claim 5, wherein the encoded data furtherincludes one or more other VLCs that each represent a transform type anda subblock pattern for one of the plural blocks of the macroblock thatis coded other than the first block.
 7. The method of claim 1, whereinthe VLC is signaled at macroblock level.
 8. The method of claim 1,further comprising selecting a VLC table from among multiple VLC tables,wherein the decoding the VLC uses the selected VLC table.
 9. The methodof claim 8, wherein the selecting the VLC table is based at least inpart upon a quantization level.
 10. The method of claim 9, wherein thequantization level is derived from one or more picture-level syntaxelements.
 11. The method of claim 8, wherein the multiple available VLCtables include a low rate table, a medium rate table, and a high ratetable.
 12. The method of claim 1, wherein the decoding the at least partof the encoded data further includes, for another block: decoding afirst code to determine a value for a DC differential for a DCcoefficient of the other block, wherein the DC differential represents adifference of the DC coefficient from a DC predictor; decoding a secondcode to determine a refinement of the value for the DC differential;reconstructing the DC differential using the value for the DCdifferential and the refinement, including adding the refinement to amultiple of the value for the DC differential, wherein the multiple ofthe value for the DC differential is determined by multiplying the valuefor the DC differential by a factor that depends on quantization stepsize.
 13. One or more computer-readable memory or storage devices havingstored thereon encoded data in a bitstream for at least part of a videosequence, the encoded data including a variable length code (VLC) for amacroblock with plural blocks, the VLC jointly representing a transformtype signal level, a transform type, and a subblock pattern, wherein theencoded data is organized to facilitate decoding, with a computer systemthat includes one or more processing units and implements a videodecoder, to reconstruct the macroblock by performing operations, theplural blocks of the macroblock including a first block that is coded,and the first block including plural subblocks, the operations includingdecoding the VLC to determine the transform type signal level, thetransform type, and the subblock pattern, wherein: the transform typeindicates a transform size that applies for the plural subblocks of thefirst block, the transform type further indicating how the first blockis split into the plural subblocks; and the subblock pattern indicateswhich of the plural subblocks of the first block have at least onenon-zero coefficient.
 14. The one or more computer-readable media ofclaim 13, wherein the transform type signal level is macroblock level,the transform type applies for each of the plural blocks of themacroblock that is coded, and the subblock pattern is for the firstblock.
 15. The one or more computer-readable media of claim 13, whereinthe transform type signal level is block level, the transform typeapplies for only the first block, and the subblock pattern is for thefirst block.
 16. The one or more computer-readable media of claim 13,wherein the VLC is signaled at macroblock level.
 17. A computer systemcomprising: a buffer, implemented with memory, configured to receive, aspart of encoded data in a bit stream for at least part of a videosequence, a variable length code (VLC) for a macroblock with pluralblocks, the VLC jointly representing a transform type signal level, atransform type, and a subblock pattern; and a video decoder, implementedwith one or more processing units, configured to perform operations todecode at least part of the encoded data to reconstruct the macroblock,the plural blocks of the macroblock including a first block that iscoded, and the first block including plural subblocks, the operationsincluding decoding the VLC to determine the transform type signal level,the transform type, and the subblock pattern, wherein: the transformtype indicates a transform size that applies for the plural subblocks ofthe first block, the transform type further indicating how the firstblock is split into the plural subblocks; and the subblock patternindicates which of the plural subblocks of the first block have at leastone non-zero coefficient.
 18. The computer system of claim 17, whereinthe transform type signal level is macroblock level, the transform typeapplies for each of the plural blocks of the macroblock that is coded,and the subblock pattern is for the first block.
 19. The computer systemof claim 17, wherein the transform type signal level is block level, thetransform type applies for only the first block, and the subblockpattern is for the first block.
 20. The computer system of claim 17,wherein the VLC is signaled at macroblock level.